Skip to content

HADOOP-19622. [3.4 Backport][ABFS][ReadAheadV2] Implement Read Buffer Manager V2 with improved aggressiveness#8175

Merged
anujmodi2021 merged 1 commit intoapache:branch-3.4from
ABFSDriver:HADOOP-19622_b3.4
Jan 13, 2026
Merged

HADOOP-19622. [3.4 Backport][ABFS][ReadAheadV2] Implement Read Buffer Manager V2 with improved aggressiveness#8175
anujmodi2021 merged 1 commit intoapache:branch-3.4from
ABFSDriver:HADOOP-19622_b3.4

Conversation

@anujmodi2021
Copy link
Contributor

@anujmodi2021 anujmodi2021 commented Jan 9, 2026

Description of PR

JIRA: https://issues.apache.org/jira/browse/HADOOP-19622

Implementing ReadBufferManagerV2 as per the new design document.
Following capabilities are added to ReadBufferManager:

  1. Configurable minimum and maximum number of prefetch threads.
  2. Configurable minimum and maximum size of cached buffer pool
  3. Dynamically adjusting thread pool size and buffer pool size based on workload requirement and resource utilization but within the limits defined by user.
  4. Mapping prefetched data to file ETag so that multiple streams reading same file can share the cache and save TPS.

For more details on design doc please refer to the design doc attached to parent JIRA: https://issues.apache.org/jira/browse/HADOOP-19596

How was this patch tested?

New tests added and existing test suite ran

@anujmodi2021
HADOOP-19622: [ABFS][ReadAheadV2] Implement Read Buffer Manager V2 wi…
f84683c 
…th improved aggressiveness (apache#7832)

Contributed by Anuj Modi
@hadoop-yetus
Copy link

hadoop-yetus commented Jan 9, 2026

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 48s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 10 new or modified test files.
_ branch-3.4 Compile Tests _
+1 💚 mvninstall 23m 47s branch-3.4 passed
+1 💚 compile 0m 21s branch-3.4 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 compile 0m 19s branch-3.4 passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 checkstyle 0m 18s branch-3.4 passed
+1 💚 mvnsite 0m 24s branch-3.4 passed
+1 💚 javadoc 0m 21s branch-3.4 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 20s branch-3.4 passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 spotbugs 0m 43s branch-3.4 passed
+1 💚 shadedclient 19m 10s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 17s the patch passed
+1 💚 compile 0m 19s the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javac 0m 19s the patch passed
+1 💚 compile 0m 16s the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 javac 0m 16s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 11s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 1 new + 5 unchanged - 9 fixed = 6 total (was 14)
+1 💚 mvnsite 0m 18s the patch passed
+1 💚 javadoc 0m 16s the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 19s the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚 spotbugs 0m 43s the patch passed
+1 💚 shadedclient 18m 51s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 4s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 25s The patch does not generate ASF License warnings.
72m 16s
Subsystem Report/Notes
Docker ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8175/1/artifact/out/Dockerfile
GITHUB PR #8175
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 5c24df522570 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.4 / f84683c
Default Java Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8175/1/testReport/
Max. process+thread count 552 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8175/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@anujmodi2021
Copy link
Contributor Author

anujmodi2021 commented Jan 9, 2026

:::: AGGREGATED TEST RESULT ::::

============================================================
HNS-OAuth-DFS

[WARNING] Tests run: 186, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 888, Failures: 0, Errors: 0, Skipped: 231
[WARNING] Tests run: 182, Failures: 0, Errors: 0, Skipped: 32
[WARNING] Tests run: 274, Failures: 0, Errors: 0, Skipped: 23

============================================================
HNS-SharedKey-DFS

[WARNING] Tests run: 186, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 888, Failures: 0, Errors: 0, Skipped: 180
[WARNING] Tests run: 182, Failures: 0, Errors: 0, Skipped: 32
[WARNING] Tests run: 274, Failures: 0, Errors: 0, Skipped: 10

============================================================
NonHNS-SharedKey-DFS

[WARNING] Tests run: 186, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 872, Failures: 0, Errors: 0, Skipped: 437
[WARNING] Tests run: 182, Failures: 0, Errors: 0, Skipped: 33
[WARNING] Tests run: 274, Failures: 0, Errors: 0, Skipped: 11

============================================================
AppendBlob-HNS-OAuth-DFS

[WARNING] Tests run: 186, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 888, Failures: 0, Errors: 0, Skipped: 242
[WARNING] Tests run: 182, Failures: 0, Errors: 0, Skipped: 56
[WARNING] Tests run: 274, Failures: 0, Errors: 0, Skipped: 23

============================================================
NonHNS-SharedKey-Blob

[WARNING] Tests run: 186, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 872, Failures: 0, Errors: 0, Skipped: 288
[WARNING] Tests run: 182, Failures: 0, Errors: 0, Skipped: 27
[WARNING] Tests run: 274, Failures: 0, Errors: 0, Skipped: 11

============================================================
NonHNS-OAuth-DFS

[WARNING] Tests run: 186, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 872, Failures: 0, Errors: 0, Skipped: 442
[WARNING] Tests run: 182, Failures: 0, Errors: 0, Skipped: 33
[WARNING] Tests run: 274, Failures: 0, Errors: 0, Skipped: 24

============================================================
NonHNS-OAuth-Blob

[WARNING] Tests run: 186, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 872, Failures: 0, Errors: 0, Skipped: 303
[WARNING] Tests run: 182, Failures: 0, Errors: 0, Skipped: 27
[WARNING] Tests run: 274, Failures: 0, Errors: 0, Skipped: 24

============================================================
AppendBlob-NonHNS-OAuth-Blob

[WARNING] Tests run: 186, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 872, Failures: 0, Errors: 0, Skipped: 354
[WARNING] Tests run: 182, Failures: 0, Errors: 0, Skipped: 51
[WARNING] Tests run: 274, Failures: 0, Errors: 0, Skipped: 24

============================================================
HNS-Oauth-DFS-IngressBlob

[WARNING] Tests run: 186, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 888, Failures: 0, Errors: 0, Skipped: 366
[WARNING] Tests run: 182, Failures: 0, Errors: 0, Skipped: 32
[WARNING] Tests run: 274, Failures: 0, Errors: 0, Skipped: 23

============================================================
NonHNS-OAuth-DFS-IngressBlob

[WARNING] Tests run: 186, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 872, Failures: 0, Errors: 0, Skipped: 439
[WARNING] Tests run: 182, Failures: 0, Errors: 0, Skipped: 33
[WARNING] Tests run: 274, Failures: 0, Errors: 0, Skipped: 24

@anujmodi2021 anujmodi2021 marked this pull request as ready for review January 9, 2026 17:08
@anujmodi2021
Copy link
Contributor Author

anujmodi2021 commented Jan 12, 2026

@steveloughran, wanted to check with you if its okay to go ahead with this merge to 3.4?
Or are we not accepting any new changes for 3.4.3 release?

@anujmodi2021
Copy link
Contributor Author

anujmodi2021 commented Jan 13, 2026

Since this change is already behind a config, disabled by default, I am goign ahead with the merge to 3.4.

@anujmodi2021 anujmodi2021 merged commit b59aa63 into apache:branch-3.4 Jan 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ABFS TOOLS trunk

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@anujmodi2021 @hadoop-yetus